A tutor receives a task; a lead designs the pipeline that decides which tasks exist, who sees them, and how their quality is proven
Day 36 of 66
For seven weeks you've sharpened the craft: rubrics, agreement, preference pairs, evals, the analysis stack. Phase 2 changes your altitude. The question is no longer "how do I label this item well?" It's "how do hundreds of items flow through a system so the labels come out good — reliably, measurably, at scale, without me watching each one?"
As a tutor, a task lands in your queue and you do it. You never chose it, never decided how many people would see it, never set what "done" means. This week you design all of that. The artifact you'll build is a pipeline — and a pipeline is not a tool. It's a set of decisions.
A pipeline is the flow an item travels from raw to trusted: raw items → sampling → queue → routing to annotators → consensus → review → final label → training set. Each arrow is a design decision you own. The tool (Label Studio, Argilla, an internal system) is just where those decisions get executed. The decisions are the job.
Walk this once slowly. Every later day in Phase 2 is a deeper look at one of these stages.
The pool of things that might need labeling — generated images, video clips, prompt/response pairs. Usually far more than you can afford to annotate. So you don't annotate all of them.
You choose which items enter the queue and in what proportion. Random is rarely right — you'll often stratify (guarantee coverage of each task type) or oversample suspected weak spots. Sampling is where you decide what the model actually learns about. (Day 37.)
The ordered work list handed to annotators. Into this queue you also secretly mix gold items — items whose correct answer you already know — so you can measure quality live without anyone gaming it. (Day 37–38.)
Not every item goes to everyone. Safety-sensitive items go to trained reviewers; multimodal items go to people with your eye for it. Routing matches item to skill.
How many people label the same item. One annotator is cheap and fast; three with a majority vote is slower and costlier but catches individual error. You scale this with difficulty and risk. (Day 37.)
When a label is uncertain, disputed, or high-stakes, it escalates for a second look. Most labels never need this; the few that do are where errors hide. (Day 39.)
The resolved, trusted label leaves the pipeline and becomes a row a model trains or is evaluated on. This is the whole point — everything upstream exists to make this row trustworthy.
Gold items — known-answer items shuffled invisibly into the work — are how you measure accuracy continuously, not just in an after-the-fact audit. If an annotator misses 3 of their last 10 gold items, you know now, before that error contaminates thousands of real labels. The catch: gold only works if annotators can't tell which items are gold. We'll design that hiding in Day 38.
Two open-source platforms dominate this space. You should know what each is for, but resist the urge to make this a tool tutorial. The design decisions transfer across any tool; the tool does not transfer your design thinking.
A flexible, open-source labeling platform. You define a labeling config (what the annotator sees and what they can input — a rating scale, a choice between two images, a bounding box), and it supports multiple annotators per task, agreement metrics, and review queues out of the box. Strong for the multimodal image/video work that is your edge. Skim the Label Studio documentation — the labeling-config and quality sections — to see how it models a task, consensus, and review.
An open-source data platform built specifically for LLM feedback and preference data. Where Label Studio is general, Argilla is purpose-built for the SFT / rating / preference shapes you studied in Weeks 4 and 6 — chosen/rejected pairs, ratings, text feedback. Skim the Argilla documentation to see how it structures preference and rating tasks.
When you interview, nobody asks "can you click around Label Studio?" They ask "how would you design the queue so we catch quality drift before it ships?" The tool is an hour of learning. The pipeline design is the skill. Learn enough of the tool to know what's possible — then spend your real energy on the consensus, gold, and review decisions that the tool merely executes.
Before the build day, get the flow out of your head and onto paper. Pick one real task you've done as a tutor — say, rating Grok image generations against a quality rubric.
Notice what happened: you just made the same calls a lead makes, on a task you know cold. The rest of the week makes each call rigorous.
A tutor does the task that lands in the queue. A lead designs the queue itself — every arrow from raw item to training set is a decision they own, and they treat Label Studio or Argilla as where those decisions get executed, not as the job.
Say this in an interview: "I don't think of an annotation system as a tool you click around — I think of it as a flow of decisions: sampling, consensus, gold, review. The platform just executes the pipeline I design."